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(54) Color image pickup apparatus 

(57) In apparatus for processing a digital color 
image value originally obtained from an image sensor or 
the like having color image pixels aligned in rows and 
columns having diagonal pixels with sufficient informa- 
tion to permit luminance or chrominance estimation the 
digital image having at least three separate color values 
but only one color value for each image pixel location, 
such apparatus computing the luminance or chromi- 
nance values for each image pixel. The apparatus 
stores the digital image and includes a processor oper- 
atively associated with the stored image and responsive 
to said stored measured digital color image values for 
computing low frequency luminance or chrominance 
values. In response to the computed low frequency 
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luminance or chrominance values the Laplao'an sec- 
ond-order and gradient values in at least two image ori- 
entations are computed from neighboring image pixels. 
The Laplacian second-order and the gradient values 
are combined to define a value for each image pixel 
such that there are sets of values with each value corre- 
sponding to a classifier and for selecting a preferred ori- 
entation from a group of orientations for the 
interpolation based upon the classifier. Finally, in 
response to the preferred orientation and the measured 
digital image for estimating the luminance or chromi- 
nance value for each corresponding image pixel. 
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Description 

[0001 ] This invention relates to the field of electronic imaging and is particularly suitable to electroni(rstiII imaging with 
improved luminance and chrominance pixel values from a measured digital image originally produced by an image sen- 
5 sor. 

[0002] In electronic color imaging, it is desirable to simultaneously capture image data in three color planes, usually 
red. green and blue, although cyan, magenta, and yellow can also be used. When the three color planes are combined, 
it is possible to create high-quality color images. Capturing these three sets of image data can be done in a number of 
ways. In electronic photography, this is sometimes accomplished by using a single two dimensional array of sensors 

10 that are covered by a pattern of red, green and blue filters. This type of sensor is known as a color filter array or CFA. 
Below is shown the red (R), green (G) and blue (B) pixels as are commonly arranged on a CFA sensor. 
[0003] When a color image is captured using a CFA, it is necessary to interpolate the red, green and blue values so 
that there is an estimate of all three color values for each sensor location. Once the interpolation is done, each picture 
element, or pixel, has three color values and can be processed by a variety of known image processing techniques 

15 depending on the needs of the system. Some examples of the reasons for processing are to do image sharpening, color 
correction or halftoning. 

[0004] The following shows how red green and blue pixels can be arranged in a particular color filter array pattern, 
hereinafter referred to as the Bayer color filter array. For a more detailed description see US-A-3,971 ,065. 
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[0005] In processing cola digital images produced by an image sensor, color specification values such as, for exam- 
30 pie, luminance and chrominance values for each pixel need to be computed. One of the problems found in processing 
color image values produced by an image sensor is that the luminance values for pixels on the edge of a feature of an 
image can be inaccurately interpolated. 

[0006] An object of this invention is to provide an improved apparatus which produces color specification values for 
pixels particularly those on the diagonal edge of a feature of the digital image. 
35 [0007] This object is achieved in apparatus for processing a digital color image value originally obtained from an 
image sensor or the like having color image pixels aligned in rows and columns having diagonal pixels with sufficient 
information to permit luminance estimation the digital image having at least three separate color values but only one 
color value for each image pixel location, such apparatus computing the luminance values for each image pixel, com- 
prising: 

AO 

means for storing the digital image; 

processor means operative with said storing means and responsive to said stored measured digital color image 
values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and gra- 

45 dient values in at least two image orientations from neighboring image pixels; 

means for combining the Laplacian second-order and the gradient values to define a value for each image pixel in 
the kernel such that there are sets of values with each value corresponding to a classifier and for selecting a pre- 
ferred orientation from a group of orientations for the interpolation based upon the classifier; and 
means responsive to the preferred orientation and the measured digital image for estimating a portion of the color 

so specification for each corresponding image pixel. 

[0008] An advantage of this invention is that it is computationally efficient both in execution time and memory storage 
requirements for producing improved luminance or chrominance values. It is another advantage of the present invention 
to use the combination of the Laplacian second-order and gradient values to produce a classifier which substantially 
55 reduces artifacts (color interpolation) in output image are substantially reduced. 

FIG. 1 is a block diagram of an electronic still camera employing interpolation processing according to the invention; 
FIG. 2 is a block diagram of the logic of the interpolation processing technique for producing luminance in accord- 
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ance with the invention; 

FIGS. 3A and 3B respectively show detailed block diagrams of the luminance values block 32 and the chrominance 
values block 34 shown in FIG. 2; 

FIG. 4 depicts a kernel used in computing low frequency luminance values; 
5 FIGS. 5A-C depict pixel weights used in computing a classifier in the classification block 48 shown in FIGS. 3A and 
3B; 

FIGS. 6A-C also depict pixel weights used in computing another classifier produced by the classification block 48; 
FIGS. 7A-7E respectively show the kernel weights for predicting luminance for pixels which have been classified as 
horizontal, vertical, flat, cfiagl, and diag2; 
10 FIG. 8A shows a prior art pixel pattern; 

FIG. 8B shows a pixel array pattern which is suitable for chrominance correction; and 
FIG. 9 depicts a block of interpolated chrominance values. 

[0009] For any given pixel, the luminance and chrominance values constitute color specification for a pixel. For pur- 
rs poses of this disclosure, the term "portion of a color specification" will refer to a pixel's luminance or chrominance val- 
ues. 

[001 0] Single-sensor electronic cameras employing color filter arrays are well known. Elements not specifically shown 
or described herein may be selected from those known in the art. 

[001 1] Referring initially to FIGS. 1 and 2, an electronic still camera 1 is divided generally into an input section 2 and 

20 an interpolation and recording section 4. The input section 2 includes an exposure section 10 for directing image light 
from a subject (not shown) toward an image sensor 12. Although not shown, the exposure section 10 includes conven- 
tional optics for directing the image light through a diaphragm, which regulates the optical aperture, and a shutter, which 
regulates exposure time. The image sensor 12, which includes a two-dimensional array of colored photosites or pixels 
corresponding to picture elements of the image, can be a conventional charge-coupled device (CCD) using either well- 

25 known interline transfer or frame transfer techniques. The image sensor 12 is covered by a color filter array (CFA) 13. 
For an example of a color filter array which is particularly suitable for use in the present invention reference is made to 
commonly-assigned US-A-5,631,703, the disclosure of which is incorporated by reference. The image sensor 12 is 
exposed to image light so that analog image charge information is generated in respective photosites. The charge infor- 
mation is applied to an output diode 14, which converts the charge information to analog image signals corresponding 

30 to respective picture elements. The analog image signals are applied to an A/D converter 16, which generates a digital 
image value from the analog input signal for each picture element The digital values are applied to an image buffer 1 8, 
which may be a random access memory (RAM) with storage capacity for a plurality of still images. 
[0012] A control processor 20 generally controls the input section 2 of the electronic still camera 1 by initiating and 
controlling exposure (by operation by the diaphragm and shutter (not shown) in the exposure section 1 0), by generating 

35 the horizontal and vertical clocks needed for driving the image sensor 12 and for clocking image information therefrom, 
and by enabling the A/D converter 16 in conjunction with the image buffer 18 for each value segment relating to a pic- 
ture element. The control processor 20 typically includes a microprocessor and appropriate memory coupled to a sys- 
tem timing circuit Once a certain number of digital image values have been accumulated in the image buffer 18, the 
stored values are applied to a digital signal processor 22, which controls the throughput processing rate for the interpo- 

40 lation and recording section 4 of the electronic still camera 1 . The digital signal processor 22 applies an interpolation 
algorithm to the digital image values, and sends the interpolated values to a conventional, removable memory card 24 
via a connector 26. Although an electronic still camera 1 has been described as including a digital signal processor, it 
will be understood that the digital signal processor 22 does not have to be an integral part of the electronic still camera 
1 . A requirement of this invention is that the digital image values are provided from an image sensor. 

45 [001 3] Since the interpolation and related processing ordinarily occurs over several steps, the intermediate products 
of the processing algorithm are stored in a processing buffer 28. The processing buffer 28 may also be configured as 
part of the memory space of the image buffer 18. The number of image values needed in the image buffer 18 before 
digital processing can begin depends on the type of processing, that is, for a neighborhood interpolation to begin, a 
block of values including at least a portion of the image values comprising a video frame must be available. Conse- 

50 quently, in most circumstances, the interpolation may commence as soon as the requisite block of picture elements is 
present in the buffer 1 8. 

[0014] The input section 2 operates at a rate commensurate with normal operation of the electronic still camera 1 
while interpolation, which may consume more time, can be relatively divorced from the input rate. The exposure section 
10 exposes the image sensor 12 to image light for a time period dependent upon exposure requirements, for example. 
55 a time period between 1/1 000 second and several seconds. The image charge is then swept from the photosites in the 
image sensor 1 2, convened to a digital format, and written into the image buffer 1 8. The driving signals provided by the 
control processor 20 to the image sensor 12, the A/D converter 16 and the buffer 18 are accordingly generated to 
achieve such a transfer. The processing throughput rate of the interpolation and recording section 4 is determined by 
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the speed of the cfigital signal processor 22. 

[001 5] One desirable consequence of this architecture is that the processing algorithm employed in the interpolation 
and recording section may be selected for quality treatment of the image rather than for throughput speed. This, of 
course, can put a delay between consecutive pictures which may affect the user, depending on the time between pho- 
5 tographic events. This is a problem since it is well known and understood in the field of electronic imaging that a digital 
still camera should provide a continuous shooting capability for a successive sequence of images. For this reason, the 
image buffer 18 shown in FIG. 1 provides for storage of a plurality of images, in effect allowing a series of images to 
"stack up" at video rates. The size of the buffer is established to hold enough consecutive images to cover most picture- 
taking situations. 

10 [0016] An operation display panel 30 is connected to the control processor 20 for displaying information useful in oper- 
ation of the electronic still camera 1 . Such information might include typical photographic data, such as shutter speed, 
aperture, exposure bias, color balance (auto, tungsten, fluorescent, daylight), field/frame, low battery, low light, expo- 
sure modes (aperture preferred, shutter preferred), and so on. Moreover, other information unique to this type of elec- 
tronic still camera 1 is displayed. For instance, the removable memory card 24 would ordinarily include a directory 

75 signifying the beginning and ending of each stored image. This would show on the display panel 30 as either (or both) 
the number of images stored or the number of image spaces remaining, or estimated to be remaining. 
[0017] The digital signal processor 22 interpolates each still video image stored in the image buffer 18 according to 
the interpolation technique shown in FIG. 2. The interpolation of missing data values at each pixel location follows the 
sequence shewn in FIG. 2, as will later be discussed. 

20 [0018] In the implementation shown in FIG. 2, the digital signal processor 22 provides an adaptive interpolation tech- 
nique to provide a compute luminance function shown as luminance values block 32 for optimizing luminance values as 
will be described hereinafter in connection with FIG. 3A. After the luminance values are computed then a chrominance 
values block 34 computes the chrominance values of each pixel based upon the computed final luminance values. 
Finally an RGB values block 36 computes the image in Red(R), Green(G), Blue(B) format which are used for an image 

25 display or for making a hard copy output Although this disclosure is in reference to computing red, green, and blue val- 
ues, it will be understood that it is also applicable to other color spaces such as cyan, magenta, and yellow. Another 
color space that can be used is to use luminance and chrominance values which are typically referred to as YCC color 
spaces. The Y refers to luminance and the two C s refer to chrominance. 

[001 9] Turning now to FIG. 3A where the luminance values block 32 is shown in more detail. A digital image from the 
30 image buffer 1 8 of FIG. 1 is applied to a blur block 42 which computes low frequency luminance values. The purpose 
of blur block 42 is to remove the high frequency components from the digital image. Reference will now be made to FIG. 
4 where a 5x5 blur kernel arrangement is shown. For any given pixel which is selected as the center pixel of a 5x5 kernel 
it is assigned value weight of 4. Pixels which surround the kernel of interest are also assigned different values. These 
values shown in FIG. 4 are representative and those skilled in the art will appreciate that other values can be selected 
35 which will also provide an appropriate blur function. Note that once the weighted sum is computed is divided by 20, 
which is the sum of all the weights in the blur kernel. The following equations depict how these kernel values are used 
to compute each pixel in a digital image which has high frequency components removed. 
[0020] If the recorded CFA color values are: 

40 A u A 12 A 13 A 14 A 15 
*21 A22A23A24A25 

A 31 A32 A33 A34 A35 
A41 A42 A^ A44 A^ 
*51 ^52 *53 *54 *55 

45 

then the following equations show how the blur kernel values of FIG. 4 are used to produce the blurred luminance value 
B33 for the center pixel above. 

50 
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B 33 = ( l*Ai 3 
5 +1*A 2 2 + 2*A23 + 1*A 2 4 

+ 1*A 31 + 2*A32 + 4*A 3 3 + 2*A34 + 1*A 3 5 
+ 1*A42 + 2*A43 + l5>A 44 
10 +1*A53 )/2Q 



The output of blur block 42 is applied to correction block 44 and classification block 48. The computation provided by 
is correction block 44 will now be described. 
[0021 ] If the blurred luminance values are: 

B 11 B 12 B 13 B 14 B 15 
B21 B22 B24 B25 
20 ^1 &32 B35 

B 41 B 42 B43 B44 B^s 
B51 B52 B53 B54 B55 

then the following equations shew how the horizontal (horz) and vertical (vert) classifiers are computed for the center 
25 pixel. 

horz = Abs( u - w ) + Abs( u - 2*v + w) 

where 

30 

the term ( u - w ) is a gradient value, 

the term ( u - 2*v + w ) is a Laplacian second-order value, 

u = B3 1t 

v=B33, 

35 W = B35. 



vert = Abs( u - w ) + Abs( u - 2*v + w) 

40 

where 

u = B 13i 
v=Bs3. 

45 W = B53. 

The two diagonal orientations will be named diagl and diag2 and correspond to a diagonal line of slope 1 (connecting 
lower left to upper right) and a diagonal line of slope -1 (connecting upper left to lower right) respectively. The classifiers 
for diagl and diag2 are computed using the kernel values shown in FIGS. 5A-5C and FIGS. 6A-6C respectively. The 
50 classifier equations using said kernel values are as follows: 

diagl = ( Abs{u-w ) + Abs( u - 2*v + w ))/3 

where 

55 
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u = ( 1*B M 

+ 1*B 2 3 + 2*B24+1*B 2 5 
+ 1 * B 34> 



10 v = ( **^23 

+ 1*632 + 2*633 + 1*634 
+ 1 * B 43) 

75 

w = ( 1*B 32 

+ 1*B 41 + 2*B 42 +1*B 43 
+ 1*B 52) 

and 

25 

diag2 = ( Abs{ u - w ) + Abs(u - 2*v + w ))/3 

where 

30 u = ( 1*B 



35 



12 

+ l*B 2 i +2*B 22 + 1*B 2 3 
+ 1*B 32) 



v = ( 1*B 23 
<o +1*B 32 + 2*B33 + 1*B 3 4 

+ 1+B 43) 



45 w = ( 1*834 



+ 1*B 4 3 + 2*B44+1*B 4 5 
+ 1 * B 54) 



50 



[0022] If a pixel's horizontal and vertical classifier are both less than a fixed threshold, such as 24, the pixel is classi- 
fied as liar and the flat predictor is used (see FIG. 7C). Otherwise, a pixel's classification is determined by the smallest 
classifier which indicates the preferred orientation of interpolation such as horizontal, vertical diagl , or diag2. 
55 [0023] The predictors for interpolating in the preferred orientations horizontal, vertical, diagl . and diag2 are shown in 
FIGS. 7A. 7B, 7D. and 7E, respectively. For a specific example, suppose the classifier values for horizontal, vertical, 
diagl. and diag2 were 32, 20, 18, and 25, respectively. Because the horizontal classifier 32 exceeds the threshold of 
24. the pixel is not classified as "flat." So then the minimum value 18 being the diagl classifier determines that the pre- 
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ferred cfirection for interpolation is diagl. Using the predictor kernel weights for the diagl direction (FIG. 7D), the pre- 
dicted luminance value V33 would be: 

V 33 = ( 1*A 14 + 

2*A 23 + 1*A 2 5 + 

2*A 32 + 2*A 33 + 2* Aw + 
10 l*A4i+ 2*A4 3 + 

!* A 52 )/14 



15 

[0024] The operation of correction block 44 will now be discussed. The luminance correction term for a pixel depends 
on its classification and the computed blurred luminance values. A pixel classified as "flat" has a zero luminance cor- 
rection. The other correction terms are determined as follows: 

20 

horz:C = (-B 31 +2*B33-B 3 5) 
vert:C = (-B 13 + 2*B33-B53) 
diagl : C = (-B 15 + 2*833 - B 51 ) 
diag2: C = (-B^ + 2*633 - B 55 ) 

25 

Whichever correction term (C) is used, it is then multiplied by a constant such as 3/8 and substituted into the following 
function: 

Max( Min( 0, x + 6), -54 ) if x < 0 

30 correction = { 

Min(Max(0,x-6),54) ifO<x 

35 where 

x = C*(3/8). 

[0025] Referring to FIG. 8B, the color filter array filters used are cyan (C), magenta (M), yellow (Y), and green (G). 
40 Luminance (V) can be defined equivalents in two ways, one using C and Y, the other using M and G. The equations are: 

V = (Y + C)/2 

and 

45 

V = (M + G)/2 
The two chrominance values Ca and Cb are defined as: 

so Ca = (Y-C)/2 

and 

Cb = (M-G)/2 

55 

From these equations it can be determined that: 

Ca = Y-V Cb = M-V 
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Ca = V-C Cb = V-G 

[0026] FIG. 3B which shows in detail the chrominance values block 34 shown in FIG. 2. Where parts correspond to 
FIG. 3A, the same numerals will be used. The color filter array data is applied to a partial block 52 which also receives 
luminance values as shown in FIG. 2. The purpose of the partial block is to provide a computation which provides 
chroma values that are completed by the complete block 54. The complete block 54 also receives an input from the 
classification block 48. 

[0027] Because the luminance interpolation has already been completed, each pixel location has an interpolated 
luminance value in addition to its original CFA color value (C, M, Y. or G). The pixels having Y or C can now use the 
interpolated luminance V in one of the above equations to compute Ca in the partial block 52. Similarly, the pixels hav- 
ing M or G can use the interpolated luminance V to compute Cb in the partial block 52. Thus, each pixel now has an 
interpolated luminance value and one interpolated chrominance value. This input is applied to the complete block 54. 
[0028] Since the M and G pixels form 2x2 sub-blocks, as do the Y and C pixels (see FIG. 8B), the chrominance values 
Ca and Cb are computed on these same sub-blocks. That means that following the array shown in FIG. 8B, the known 
Ca and Cb values, computed in partial block 52, are arranged as follows: 

Cb Cb Ca Ca Cb Cb 
Cb Cb Ca Ca Cb Cb 
Ca Ca Cb Cb Ca Ca 
Ca Ca Cb Cb Ca Ca 
Cb Cb Ca Ca Cb Cb 
Cb Cb Ca Ca Cb Cb 

[0029] Referring to FIG. 9, which illustrates the operation of the complete block 54, the upper left 5x5 block of this 
array of chrominance values is shown. The center pixel already has an interpolated value for Cb and needs to get an 
interpolated value for Ca. Any pixel needing a value of Ca has a neighbor to the left (one or two steps away), a neighbor 
to the right (one or two steps away), a neighbor above (one or two steps away), a neighbor below (one or two steps 
away), and two neighbors touching on opposite corners (either top left and bottom right, or top right and bottom left) and 
all having an interpolated value for Ca. If the pixel was originally classified as flat or horizontal, the two values in the 
horizontal direction (at the right and left neighbors) are averaged and become the interpolated value for Ca. If the pixel 
was classified as vertical, then the values of Ca above and below are averaged to become the interpolated value for Ca. 
If the pixel was classified either as cfiagl or diag2, the two values from opposite corners are averaged to become the 
interpolated value for Ca. 

[0030] For example, if the center pixel (FIG. 9) were classified flat or horizontal the interpolation equation would be: 

Ca^ = (Ca 32 + Ca 35 )/2 
If the center pixel were classified as vertical the interpolation equation would be: 

Ca 3 3 = (Ca 2 3 + Ca 53 )/2 
ff the center pixel were classified as diagl or diag2 the interpolation equation would be: 

Ca33=(Ca 24 + Ca 42 )/2 

[0031] Any other pixel needing an interpolated value for Ca may be processed in a similar manner. Furthermore, each 
pixel needing an interpolated value for Cb may also be processed in a similar way with the roles of Ca and Cb reversed. 
When this step is done, every pixel has a luminance and two chrominance values, and the interpolation process is com- 
plete. 

[0032] For each pixel the conversion to red, green, and blue values (see RGB values block 36, FIG. 2) is accomplished 
by the following equations: 
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R + V + 2*Ca + Cb 
0 + V -Cb 

5 

B + V-2*Ca + Cb 



10 [0033] The present invention is applicable to a digital color image value originally obtained from an image sensor or 
the like and has color image pixels aligned in rows and columns to define repeating kernels having diagonal pixels with 
sufficient information to permit luminance estimation the digital image having at least three separate color values but 
only one color value for each image pixel location. For example, the so-called Bayer array generally takes the format 
shown in FIG. 8A. With this format, there is insufficient luminance (that is, green) information along certain diagonals 

75 labeled as b, d, and f of FIG. 8A. With other diagonals, there is corrpfete information. They are labeled a, c, and e. The 
array pattern shown in US-A-5,631 ,703 has sufficient information (that is, magenta and green values or cyan and yellow 
values) along all its diagonals (for example, a-e of FIG. 8B) and so is especially suitable for use with the present inven- 
tion. 

[0034] The present invention can be embodied in a computer program stored on a computer readable product such 
20 as, for example, magnetic storage media, such as a magnetic disk (for example, a floppy disk), magnetic tape, optical 
disks, optical tape, or machine readable memory. 
[0035] Other features of the invention are included below. 

[0036] In apparatus for processing a digital color image value originally obtained from an image sensor or the like hav- 
ing color image pixels aligned in rows and columns having diagonal pixels with sufficient information to permit lumi- 
25 nance estimation the digital image having at least three separate color values but only one color value for each image 
pixel location, such apparatus computing the luminance values for each image pixel, comprising: 

means for storing the digital image; 

processor means operative with said storing means and responsive to said stored measured digital color image 
30 values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and gra- 
dient values in at least two image orientations from neighboring image pixels; 

means for combining the Laplacian second-order and the gradient values to define a value for each image pixel in 
the kernel such that there are sets of values with each value corresponding to a classifier and for selecting a pre- 
35 ferred orientation from a group of orientations for the interpolation based upon the classifier; and 

means responsive to the preferred orientation and the measured digital image for estimating the chrominance 
value for each corresponding image pixel. 

[0037] The apparatus wherein there are at least two possible selected orientations which include horizontal and ver- 
40 tical. 

[0038] The apparatus wherein there at least four possible selected orientations which include horizontal, vertical, and 
first and second diagonals. 

[0039] The apparatus further including means for selecting the smallest classifier which determines the selected ori- 
entation and corresponds to the predictor. 
45 [0040] In apparatus for processing a measured digital color image value originally obtained from an image sensor 
having cola image pixels aligned in rows and columns the image sensor generating at least three separate color values 
but only one color value for each image pixel location, such apparatus computing the luminance values for each image 
pixel, comprising: 

so means for storing the measured digital color image value; 

processor means operative with said storing means and responsive to said stored measured digital color image 
values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and gra- 
dient values in at least two image orientations from neighboring image pixels; 
55 means for combining the Laplacian second-order and the gradient values to define a value for each image pixel 
such that there are sets of values with each value corresponding to a classifier and for selecting a preferred orien- 
tation from a group of orientations including flat, horizontal, vertical, and first and second diagonal orientations for 
the interpolation based upon the classifier; and 
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means responsive to the preferred orientation and the measured digital image for determining the chrominance 
value for each corresponding image pixel. 

[0041 ] The apparatus further inducing means for using the smallest classifiers and when the horizontal and vertical 
5 classifiers are below a predetermined threshold level for selecting the flat orientation and when the smallest classifier 
corresponds to the horizontal, vertical, and first and second diagonals for selecting one of those orientations as the pre- 
ferred orientation. 

[0042] The apparatus including means responsive to the selected orientation being flat to compute the average of the 
horizontal and vertical for computing a predictor which is used in determining the chrominance value for each corre- 
w sponcfing image pixel. 

[0043] In apparatus for processing a measured digital color image value originally obtained from an image sensor 
having color image pixels aligned in rows and columns the image sensor generating at least three separate color values 
but only one color value for each image pixel location, such apparatus computing the chrominance values for each 
image pixel, comprising: 

15 

means for storing the measured digital color image value; 

processor means operative with said storing means and responsive to said stored measured digital color image 
values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and gra- 

20 dient values in at least two image orientations from neighboring image pixels; 

means for adding the Laplacian second-order and the gradient values to define a value for each image pixel such 
that there are sets of values with each value corresponding to a classifier and for selecting a preferred orientation 
from a group of orientations including flat, horizontal, vertical, and first and second diagonal orientations for the 
interpolation based upon the classifier; and 

25 means responsive to the preferred orientation and the measured digital image for determining the chrominance 
value fa each corresponding image pixel. 

[0044] A computer program, product, comprising a computer readable storage medium including program means 
comprising: 

30 

a) means for processing a digital color image value originally obtained from an image sensor or the like having color 
image pixels aligned in rows and columns t having diagonal pixels with sufficient information to permit luminance 
estimation the digital image having at least three separate color values but only one color value for each image pixel 
location; and 

35 b) means for computing the chrominance values for each image pixel, including: 

i) means for storing the digital image; 

ii) processor means operative with said storing means and responsive to said stored measured digital color 
image values for computing low frequency luminance values; 

40 iii) means responsive to the computed low frequency luminance values for obtaining Laplacian second-order 

and gradient values in at least two image orientations from neighboring image pixels; 
iv) means for combining the Laplacian second-order and the gradient values to define a value for each image 
pixel in the kernel such that there are sets of values with each value corresponding to a classifier and for select- 
ing a preferred orientation from a group of orientations for the interpolation based upon the classifier; and 

45 v) means responsive to the preferred orientation and the measured digital image for estimating the chromi- 

nance value for each corresponding image pixel. 

PARTS UST 

50 [0045] 

2 input section 

4 recording section 

10 exposure section 

55 12 image sensor 

13 color filter array 

14 output diode 
16 A/D converter 
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1ft 


I may 6 PUTTcl 


20 


control processor 


22 


digital signal processor 


24 


removable memory card 


26 


connector 


28 


processing buffer 


30 


display panel 


32 


luminance values block 


36 


luminance section 


38 


chroma section 


40 


select best luma interpolation 


42 


blur block 


44 


correction block 


46 


luminance block 


48 


classification block 


52 


partial block 


54 


complete block 


Claims 



1 . In apparatus for processing a digital color image value originally obtained from an image sensor or the like having 
color image pixels aligned in rows and columns having diagonal pixels with sufficient information to permit lumi- 
nance estimation the digital image having at least three separate color values but only one color value for each 
image pixel location, such apparatus computing the luminance values for each image pixel, comprising: 

means for storing the digital image; 

processor means operative with said storing means and responsive to said stored measured digital color 
image values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and 
gradient values in at least two image orientations from neighboring image pixels; 

means for combining the Laplacian second-order and the gradient values to define a value for each image pixel 
in the kernel such that there are sets of values with each value corresponding to a classifier and for selecting 
a preferred orientation from a group of orientations for the interpolation based upon the classifier; and 
means responsive to the preferred orientation and the measured digital image for estimating a portion of the 
color specification for each corresponding image pixel. 

2. In apparatus for processing a digital color image value originally obtained from an image sensor or the like having 
color image pixels aligned in rows and columns having diagonal pixels with sufficient information to permit lumi- 
nance estimation the digital image having at least three separate color values but only one color value for each 
image pixel location, such apparatus computing the luminance values for each image pixel, comprising: 

means for storing the digital image; 

processor means operative with said storing means and responsive to said stored measured digital color 
image values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and 
gradient values in at least two image orientations from neighboring image pixels; 

means for combining the Laplacian second-order and the gradient values to define a value for each image pixel 
such that there are sets of values with each value corresponding to a classifier and for selecting a preferred 
orientation from a group of orientations for the interpolation based upon the classifier; and 
means responsive to the preferred orientation and the measured digital image for estimating the luminance 
value for each corresponding image pixel. 

3. The apparatus of claim 1 wherein there are at least two possible selected orientations which include horizontal and 
vertical. 

4. The apparatus of daim 1 wherein there at least four possible selected orientations which include horizontal, verti- 
cal, and first and second diagonals. 
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5. The apparatus of claim 4 further including means for selecting the smallest classifier which determines the selected 
orientation and corresponds to the predictor. 

6. In apparatus for processing a measured digital color image value originally obtained from an image sensor having 
5 color image pixels aligned in rows and columns the image sensor generating at least three separate color values 

but only one color value for each image pixel location, such apparatus computing the luminance values for each 
image pixel, comprising: 

means for storing the measured digital color image value; 
10 processor means operative with said storing means and responsive to said stored measured digital color 

image values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and 
gradient values in at least two image orientations from neighboring image pixels; 

means for combining the Laplacian second-order and the gradient values to define a value for each image pixel 
is such that there are sets of values with each value corresponding to a classifier and for selecting a preferred 

orientation from a group of orientations including flat, horizontal, vertical, and first and second diagonal orien- 
tations for the interpolation based upon the classifier; and 

means responsive to the preferred orientation and the measured digital image for determining the luminance 
value for each corresponding image pixel. 

20 

7. The apparatus of claim 6 further including means for using the smallest classifiers and when the horizontal and ver- 
tical classifiers are below a predetermined threshold level for selecting the flat orientation and when the smallest 
classifier corresponds to the horizontal, vertical, and first and second diagonals for selecting one of those orienta- 
tions as the preferred orientation. 

25 

8. The apparatus according to claim 7 including means responsive to the selected orientation being flat to compute 
the average of the horizontal and vertical for computing a predictor which is used in determining the luminance 
value for each corresponding image pixel. 

30 9. In apparatus for processing a measured digital color image value originally obtained from an image sensor having 
color image pixels aligned in rows and columns the image sensor generating at least three separate color values 
but only one color value for each image pixel location, such apparatus computing the luminance values for each 
image pixel, comprising: 

35 means for storing the measured digital color image value; 

processor means operative with said storing means and responsive to said stored measured digital color 
image values for computing low frequency luminance values; 

means responsive to the computed low frequency luminance values for obtaining Laplacian second-order and 
gradient values in at least two image orientations from neighboring image pixels; 
40 means for adding the Laplacian second-order and the gradient values to define a value for each image pixel 

such that there are sets of values with each value corresponding to a classifier and for selecting a preferred 
orientation from a group of orientations including flat horizontal, vertical, and first and second diagonal orien- 
tations for the interpolation based upon the classifier; and 

means responsive to the preferred orientation and the measured digital image for determining the luminance 
45 value for each corresponcfing image pixel. 

10. A computer program, product comprising a computer readable storage medium including program means com- 
prising: 

so a) means for processing a digital color image value originally obtained from an image sensor or the like having 

color image pixels aligned in rows and columns having diagonal pixels with sufficient information to permit lumi- 
nance estimation the digital image having at least three separate color values but only one color value for each 
image pixel location; and 

b) means for computing the luminance values for each image pixel, including: 

55 

i) means for storing the digital image; 

ii) processor means operative with said storing means and responsive to said stored measured digital 
color image values for computing lew frequency luminance values; 
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Hi) means responsive to the computed low frequency luminance values for obtaining Laplacian second- 
order and gradient values in at least two image orientations from neighboring image pixels; 

iv) means for combining the Laplacian second-order and the gradient values to define a value for each 
image pixel in the kernel such that there are sets of values with each value corresponding to a classifier 
and for selecting a preferred orientation from a group of orientations for the interpolation based upon the 
classifier; and 

v) means responsive to the preferred orientation and the measured digital image for estimating the lumi- 
nance value for each corresponding image pixel. 
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